{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b35d2e8f",
   "metadata": {},
   "source": [
    "# Comp2-2 临床类型数据\n",
    "\n",
    "主要适配于临床数据的建模和刻画。典型的应用场景探究rad_score最终临床诊断的作用。\n",
    "\n",
    "## Onekey步骤\n",
    "\n",
    "1. 数据校验，检查数据格式是否正确。\n",
    "3. 查看一些统计信息，检查数据时候存在异常点。\n",
    "4. 正则化，将数据变化到服从 N~(0, 1)。\n",
    "5. 通过相关系数，例如spearman、person等筛选出特征。\n",
    "6. 构建训练集和测试集，这里使用的是随机划分，正常多中心验证，需要大家根据自己的场景构建两份数据。\n",
    "7. 通过Lasso筛选特征，选取其中的非0项作为后续模型的特征。\n",
    "8. 使用机器学习算法，例如LR、SVM、RF等进行任务学习。\n",
    "9. 模型结果可视化，例如AUC、ROC曲线，混淆矩阵等。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb054795",
   "metadata": {},
   "source": [
    "## 一、数据校验\n",
    "首先需要检查诊断数据，如果显示`检查通过！`择可以正常运行之后的，否则请根据提示调整数据。\n",
    "\n",
    "数据文件中的数据都是数值类型，或者可以映射成数值类型，这里的`label`某些情况下可能是非数值的，需要自定义数值函数。\n",
    "\n",
    "**注意：在使用树模型时，可以存在缺失，但是线性模型不允许缺失，请自行根据需要填充缺省值**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a4cbe42",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import onekey_algo.custom.components as okcomp\n",
    "from onekey_algo import OnekeyDS as okds\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# 读取数据，B超诊断阳性=1，bc_data.csv是要读取的数据。\n",
    "data_file = okds.phy_bio\n",
    "labels = ['生存时间（月）']\n",
    "\n",
    "structed_data = pd.read_csv(data_file, header=0)\n",
    "# 生成1年、3年、5年生存情况\n",
    "structed_data['1year survival'] = (structed_data[labels] > 12).astype(int)\n",
    "structed_data['3year survival'] = (structed_data[labels] > 36).astype(int)\n",
    "structed_data['5year survival'] = (structed_data[labels] > 60).astype(int)\n",
    "# 重新生成labels\n",
    "labels = ['1year survival', '3year survival', '5year survival']\n",
    "\n",
    "structed_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef229b5",
   "metadata": {},
   "source": [
    "### 特征维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1de1e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删掉ID这一列。\n",
    "ids = structed_data['ID']\n",
    "structed_data = structed_data.drop(['ID', '生存时间（月）'], axis=1)\n",
    "structed_data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f8b1f6e",
   "metadata": {},
   "source": [
    "## 二、数据统计\n",
    "\n",
    "1. count，统计样本个数。\n",
    "2. mean、std, 对应特征的均值、方差\n",
    "3. min, 25%, 50%, 75%, max，对应特征的最小值，25,50,75分位数，最大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa5c306d",
   "metadata": {},
   "outputs": [],
   "source": [
    "structed_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a13a21",
   "metadata": {},
   "source": [
    "## 三、正则化\n",
    "\n",
    "`normalize_df` 为onekey中正则化的API，将数据变化到0均值1方差。正则化的方法为\n",
    "\n",
    "$column = \\frac{column - mean}{std}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33fe3d76",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import normalize_df\n",
    "data = normalize_df(structed_data, not_norm=labels)\n",
    "data = data.dropna(axis=1)\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd361f98",
   "metadata": {},
   "source": [
    "## 四、相关系数\n",
    "\n",
    "计算相关系数的方法有3种可供选择\n",
    "1. pearson （皮尔逊相关系数）: standard correlation coefficient\n",
    "\n",
    "2. kendall (肯德尔相关性系数) : Kendall Tau correlation coefficient\n",
    "\n",
    "3. spearman (斯皮尔曼相关性系数): Spearman rank correlation\n",
    "\n",
    "三种相关系数参考：https://blog.csdn.net/zmqsdu9001/article/details/82840332"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "979a9aeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_corr = data.corr('pearson')\n",
    "kendall_corr = data.corr('kendall')\n",
    "spearman_corr = data.corr('spearman')\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from onekey_algo.custom.components.comp1 import draw_matrix\n",
    "import os\n",
    "\n",
    "os.makedirs('img', exist_ok=True)\n",
    "os.makedirs('results', exist_ok=True)\n",
    "\n",
    "plt.figure(figsize=(15.0, 12.0))\n",
    "plt.rcParams['font.sans-serif'] = 'SimHei'\n",
    "plt.rcParams['axes.unicode_minus'] = False\n",
    "\n",
    "# 选择可视化的相关系数\n",
    "draw_matrix(pearson_corr, annot=True, cmap='YlGnBu', cbar=False)\n",
    "plt.savefig(f'img/features_corr.svg', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fcb5c23",
   "metadata": {},
   "source": [
    "### 特征筛选\n",
    "\n",
    "根据相关系数，对于相关性比较高的特征（一般文献取corr>0.9），两者保留其一。\n",
    "\n",
    "```python\n",
    "def select_feature(corr, threshold: float = 0.9, keep: int = 1):\n",
    "    \"\"\"\n",
    "    * corr, 相关系数矩阵。\n",
    "    * threshold，筛选的相关系数的阈值，大于阈值的两者保留其一（可以根据keep修改，可以是其二...）。默认阈值为0.9\n",
    "    * keep，可以选择大于相关系数，保留几个，默认只保留一个。\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fe3df7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import select_feature\n",
    "sel_feature = select_feature(spearman_corr)\n",
    "sel_feature"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e88423c1",
   "metadata": {},
   "source": [
    "### 过滤特征\n",
    "\n",
    "通过`sel_feature`过滤出筛选出来的特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86e09b3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_data = data[sel_feature]\n",
    "sel_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8e31ba9",
   "metadata": {},
   "source": [
    "## 五、构建数据\n",
    "\n",
    "将样本的训练数据X与监督信息y分离出来，并且对训练数据进行划分，一般的划分原则为80%-20%\n",
    "\n",
    "**注意1：树模型允许缺省值，如果要使用线性模型，需要先`df = df.fillna(0)`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a923e73",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import onekey_algo.custom.components as okcomp\n",
    "\n",
    "n_classes = 2\n",
    "y_data = data[labels]\n",
    "X_data = data.drop(labels, axis=1)\n",
    "# 如果要使用线性模型，需要把缺省值进行填充。\n",
    "X_data = X_data.fillna(0)\n",
    "\n",
    "X_train, X_test, y_train, y_test = okcomp.comp1.split_dataset(X_data, y_data, test_size=0.2)\n",
    "print(f\"训练集样本数：{X_train.shape}, 验证集样本数：{X_test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b7e7da",
   "metadata": {},
   "source": [
    "## 六、模型筛选\n",
    "\n",
    "根据筛选出来的数据，做模型的初步选择。当前主要使用到的是Onekey中的\n",
    "\n",
    "1. SVM，支持向量机，引用参考。\n",
    "2. KNN，K紧邻，引用参考。\n",
    "3. Decision Tree，决策树，引用参考。\n",
    "4. Random Forests, 随机森林，引用参考。\n",
    "5. XGBoost, bosting方法。引用参考。\n",
    "6. LightGBM, bosting方法，引用参考。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbab32fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_names = ['SVM', 'KNN', 'RandomForest', 'ExtraTrees', 'XGBoost', 'LightGBM']\n",
    "# model_names = ['LightGBM', 'XGBoost']\n",
    "models = okcomp.comp1.create_clf_model(model_names)\n",
    "model_names = list(models.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c369ea2d",
   "metadata": {},
   "source": [
    "### 交叉验证\n",
    "\n",
    "`n_trails`指定随机次数，每次采用的是80%训练，随机20%进行测试，找到最好的模型，以及对应的最好的数据划分。\n",
    "\n",
    "这里的数据并没有使用前面`Lasso`筛选出来的特征进行训练，理论来说，特征筛选仅对线性模型有一定作用，例如`SVM`、`LR`，但是对树模型没什么作用，例如`DecisionTree`、`Random`这些。所以默认不筛选。\n",
    "\n",
    "\n",
    "```python\n",
    "def get_bst_split(X_data: pd.DataFrame, y_data: pd.DataFrame,\n",
    "            models: dict, test_size=0.2, metric_fn=accuracy_score, n_trails=10,\n",
    "            cv: bool = False, shuffle: bool = False, metric_cut_off: float = None, random_state=None):\n",
    "    \"\"\"\n",
    "    寻找数据集中最好的数据划分。\n",
    "    Args:\n",
    "        X_data: 训练数据\n",
    "        y_data: 监督数据\n",
    "        models: 模型名称，Dict类型、\n",
    "        test_size: 测试集比例\n",
    "        metric_fn: 评价模型好坏的函数，默认准确率，可选roc_auc_score。\n",
    "        n_trails: 尝试多少次寻找最佳数据集划分。\n",
    "        cv: 是否是交叉验证，默认是False，当为True时，n_trails为交叉验证的n_fold\n",
    "        shuffle: 是否进行随机打乱\n",
    "        metric_cut_off: 当metric_fn的值达到多少时进行截断。\n",
    "        random_state: 随机种子\n",
    "\n",
    "    Returns: {'max_idx': max_idx, \"max_model\": max_model, \"max_metric\": max_metric, \"results\": results}\n",
    "\n",
    "    \"\"\"\n",
    "```\n",
    "\n",
    "**注意：这里采用了【挑数据】，如果想要严谨，请修改`n_trails=1`。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bedcf86",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# 随机使用n_trails次数据划分，找到最好的一次划分方法，并且保存在results中。\n",
    "bst_split_data = {}\n",
    "for label in labels:\n",
    "    results = okcomp.comp1.get_bst_split(X_data, y_data[label], models, test_size=0.2, \n",
    "                                         metric_fn=accuracy_score, n_trails=5, cv=True, random_state=0)\n",
    "    _, (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = results['results'][results['max_idx']]\n",
    "    bst_split_data[label] = (X_train_sel, X_test_sel, y_train_sel, y_test_sel)\n",
    "    trails, _ = zip(*results['results'])\n",
    "    cv_results = pd.DataFrame(trails, columns=model_names)\n",
    "    # 可视化每个模型在不同的数据划分中的效果。\n",
    "    sns.boxplot(data=cv_results)\n",
    "    plt.ylabel('Accuracy %')\n",
    "    plt.xlabel('Model Nmae')\n",
    "    plt.title(f'Task: {label}')\n",
    "    plt.savefig(f'img/{label}_model_cv.svg', bbox_inches = 'tight')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac7440e",
   "metadata": {},
   "source": [
    "### 模型筛选\n",
    "\n",
    "使用最好的数据划分，进行后续的模型研究。\n",
    "\n",
    "**注意**: 一般情况下论文使用的是随机划分的数据，但也有些论文使用【刻意】筛选的数据划分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7158f70a",
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = []\n",
    "for l in labels:\n",
    "    (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[l]\n",
    "    new_models = list(okcomp.comp1.create_clf_model(model_names).values())\n",
    "    for m in new_models:\n",
    "        m.fit(X_train_sel, y_train_sel)\n",
    "        y_pred = m.predict(X_test_sel)\n",
    "    targets.append(new_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88194089",
   "metadata": {},
   "source": [
    "## 七、预测结果\n",
    "\n",
    "* predictions，二维数据，每个label对应的每个模型的预测结果。\n",
    "* pred_scores，二维数据，每个label对应的每个模型的预测概率值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b53a19bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from onekey_algo.custom.components.delong import calc_95_CI\n",
    "from onekey_algo.custom.components.metrics import analysis_pred_binary\n",
    "\n",
    "predictions = []\n",
    "pred_scores = []\n",
    "for label, target in zip(labels, targets):\n",
    "    (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "    predictions.append([(model.predict(X_train_sel), model.predict(X_test_sel)) for model in target])\n",
    "    pred_scores.append([(model.predict_proba(X_train_sel), model.predict_proba(X_test_sel)) for model in target])\n",
    "\n",
    "metric = []\n",
    "pred_sel_idx = []\n",
    "for label, prediction, scores in zip(labels, predictions, pred_scores):\n",
    "    pred_sel_idx_label = []\n",
    "    (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "    for mname, (train_pred, test_pred), (train_score, test_score) in zip(model_names, prediction, scores):\n",
    "        # 计算训练集指数\n",
    "        acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres = analysis_pred_binary(y_train_sel[label], \n",
    "                                                                                              train_score[:, 1])\n",
    "        ci = f\"{ci[0]:.4f} - {ci[1]:.4f}\"\n",
    "        metric.append((mname, acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres, f\"{label}-train\"))\n",
    "                 \n",
    "        # 计算验证集指标\n",
    "        acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres = analysis_pred_binary(y_test_sel[label], \n",
    "                                                                                              test_score[:, 1])\n",
    "        ci = f\"{ci[0]:.4f} - {ci[1]:.4f}\"\n",
    "        metric.append((mname, acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres, f\"{label}-test\"))\n",
    "        # 计算thres对应的sel idx\n",
    "        pred_sel_idx_label.append(np.logical_or(test_score[:, 0] >= thres, test_score[:, 1] >= thres))\n",
    "    \n",
    "    pred_sel_idx.append(pred_sel_idx_label)\n",
    "metric = pd.DataFrame(metric, index=None, columns=['model_name', 'Accuracy', 'AUC', '95% CI',\n",
    "                                                   'Sensitivity', 'Specificity', \n",
    "                                                   'PPV', 'NPV', 'Precision', 'Recall', 'F1',\n",
    "                                                   'Threshold', 'Task'])\n",
    "metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50bc4d4",
   "metadata": {},
   "source": [
    "### 绘制曲线\n",
    "\n",
    "绘制的不同模型的准确率柱状图和折线图曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71f86dcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set_theme(style=\"whitegrid\")\n",
    "\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.subplot(211)\n",
    "sns.barplot(x='model_name', y='Accuracy', data=metric, hue='Task')\n",
    "plt.subplot(212)\n",
    "sns.lineplot(x='model_name', y='Accuracy', data=metric, hue='Task')\n",
    "plt.savefig(f'img/model_acc.svg', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f5585c2",
   "metadata": {},
   "source": [
    "## 绘制ROC曲线\n",
    "确定最好的模型，并且绘制曲线。\n",
    "\n",
    "```python\n",
    "def draw_roc(y_test, y_score, title='ROC', labels=None):\n",
    "```\n",
    "\n",
    "`sel_model = ['SVM', 'KNN']`参数为想要绘制的模型对应的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b610f5d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_model = model_names\n",
    "\n",
    "for sm in sel_model:\n",
    "    if sm in model_names:\n",
    "        sel_model_idx = model_names.index(sm)\n",
    "    \n",
    "        # Plot all ROC curves\n",
    "        for pred_score, label in zip(pred_scores, labels):\n",
    "            (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "            okcomp.comp1.draw_roc([np.array(y_train_sel[label]), np.array(y_test_sel[label])], \n",
    "                                  pred_score[sel_model_idx], \n",
    "                                  labels=['Train', 'Test'], title=f\"{label} Model: {sm}\")\n",
    "            plt.savefig(f'img/{label}_model_{sm}_roc.svg', bbox_inches = 'tight')\n",
    "            plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95c4ac53",
   "metadata": {},
   "source": [
    "### 汇总所有模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4d79e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_model = model_names\n",
    "\n",
    "for pred_score, label in zip(pred_scores, labels):\n",
    "    (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "    pred_test_scores = []\n",
    "    for sm in sel_model:\n",
    "        if sm in model_names:\n",
    "            sel_model_idx = model_names.index(sm)\n",
    "            pred_test_scores.append(pred_score[sel_model_idx][1])\n",
    "    # Plot all ROC curves\n",
    "    plt.figure(figsize=(8, 8))\n",
    "    okcomp.comp1.draw_roc([np.array(y_test_sel[label])] * len(pred_test_scores), \n",
    "                          pred_test_scores, \n",
    "                          labels=sel_model, title=f\"{label} ROC\")\n",
    "    plt.savefig(f'img/{label}_model_roc.svg', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0a57a8b",
   "metadata": {},
   "source": [
    "### DCA 决策曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46d13913",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import plot_DCA\n",
    "\n",
    "for pred_score, label in zip(pred_scores, labels):\n",
    "    (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "    pred_test_scores = []\n",
    "    for sm in sel_model:\n",
    "        if sm in model_names:\n",
    "            sel_model_idx = model_names.index(sm)\n",
    "            okcomp.comp1.plot_DCA(pred_score[sel_model_idx][1][:,1], np.array(y_test_sel[label]),\n",
    "                                  title=f'{label} Model {sm} DCA')\n",
    "            plt.savefig(f'img/{label}_model_{sm}_dca.svg', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ae6a46",
   "metadata": {},
   "source": [
    "## 绘制混淆矩阵\n",
    "\n",
    "绘制混淆矩阵，[混淆矩阵解释](https://baike.baidu.com/item/%E6%B7%B7%E6%B7%86%E7%9F%A9%E9%98%B5/10087822?fr=aladdin)\n",
    "`sel_model = ['SVM', 'KNN']`参数为想要绘制的模型对应的参数。\n",
    "\n",
    "如果需要修改标签到名称的映射，修改`class_mapping={1:'1', 0:'0'}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5012998",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置绘制参数\n",
    "sel_model = model_names\n",
    "c_matrix = {}\n",
    "\n",
    "for sm in sel_model:\n",
    "    if sm in model_names:\n",
    "        sel_model_idx = model_names.index(sm)\n",
    "        for idx, label in enumerate(labels):\n",
    "            (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "            gt = np.squeeze(bst_split_data[label][-1].values).tolist()\n",
    "            cm = okcomp.comp1.calc_confusion_matrix(predictions[idx][sel_model_idx][1], \n",
    "                                                    gt,\n",
    "                                                    class_mapping={1:'1', 0:'0'}, num_classes=2)\n",
    "            c_matrix[label] = cm\n",
    "            plt.figure(figsize=(5, 4))\n",
    "            plt.title(f'Model:{sm} for {label}')\n",
    "            okcomp.comp1.draw_matrix(cm, norm=False, annot=True, cmap='Blues', fmt='.3g')\n",
    "            plt.savefig(f'img/{label}_model_{sm}_cm.svg', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9034af0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "sel_model = model_names\n",
    "\n",
    "for idx, label in enumerate(labels):\n",
    "    for sm in sel_model:\n",
    "        if sm in model_names:\n",
    "            sel_model_idx = model_names.index(sm)\n",
    "            target = targets[idx][sel_model_idx]\n",
    "            (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = bst_split_data[label]\n",
    "            # 预测训练集和测试集数据。\n",
    "            train_indexes = np.reshape(np.array(ids.loc[list(X_train_sel.index)]), (-1, 1)).astype(str)\n",
    "            test_indexes = np.reshape(np.array(ids.loc[list(X_test_sel.index)]), (-1, 1)).astype(str)\n",
    "            y_train_pred_scores = target.predict_proba(X_train_sel)\n",
    "            y_test_pred_scores = target.predict_proba(X_test_sel)\n",
    "            columns = ['ID'] + [f\"{label}-{i}\"for i in range(y_test_pred_scores.shape[1])]\n",
    "            # 保存预测的训练集和测试集结果\n",
    "            result_train = pd.DataFrame(np.concatenate([train_indexes, y_train_pred_scores], axis=1), columns=columns)\n",
    "            result_train.to_csv(f'results/{sm}_{label}_train.csv', index=False)\n",
    "            result_test = pd.DataFrame(np.concatenate([test_indexes, y_test_pred_scores], axis=1), columns=columns)\n",
    "            result_test.to_csv(f'results/{sm}_{label}_test.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7cd4698",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
